{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping With Beautiful Soup in Python\n",
    "\n",
    "Web Scraping is a technique employed to extract large amounts of data from websites whereby the data is retrieved and saved to a local file in your computer or to a database in a table (spreadsheet) format. This process of web scraping can be a time-consuming as it requires us to know the layout of the document and isolate the items. However, Python users have access to a robust library that makes the process of web scraping easy & quick. The name of this library is `BeautifulSoup` and comes preinstalled on the Anaconda platform.\n",
    "\n",
    "In this tutorial, we will explore how to use this library to scrape a Wikipedia page for data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Tip:</b> If you do not have `BeautifulSoup` installed on Python simply run the follwing pip command `pip install beautifulsoup4`.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section One: Getting the Web Page Content\n",
    "\n",
    "The first thing we will do is import the libraries and modules. The most obvious one is the `BeautifulSoup` library as this is the library we will be using to scrape the page. However, to get the HTML content of the web page, we will need one more library; the `urllib.request` Library. This library is used to open the URL we want to scrape and then pass the HTML content of that page into the Beautiful Soup parser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# beautiful soup libraries\n",
    "from bs4 import BeautifulSoup\n",
    "from urllib.request import Request, urlopen\n",
    "import urllib.request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our libraries & modules imported, let's get started by defining the webpage URL & then requesting it. Once we have our request object, we can pass it through our BeautifulSoup object, while making sure to specify the `html.parser` as the content of the response object is HTML. The loading process is the real beauty of the `BeautifulSoup` library, **as it handles all the content for use behind the scenes.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define & request the url that we want to scrape\n",
    "wiki_url = r\"https://en.wikipedia.org/wiki/Eastern_Front_(World_War_II)\"\n",
    "html_content = urllib.request.urlopen(wiki_url)\n",
    "\n",
    "# Pass the html_content(the webpage) through our beautiful soup object\n",
    "soup = BeautifulSoup(html_content, 'html.parser')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the content is loaded, we have a new BeautifulSoup object that we can easily traverse. Here is a simple example where we use the `find` method to find the first `a` tag in our HTML code. Now, we can make our search a little more narrow by defining an attribute we want to identify. In this example, I want the `href` attribute to exist in my `a` tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a href=\"/wiki/Wikipedia:Protection_policy#semi\" title=\"This article is semi-protected until June 21, 2019 at 18:17 UTC.\"><img alt=\"Page semi-protected\" data-file-height=\"512\" data-file-width=\"512\" decoding=\"async\" height=\"20\" src=\"//upload.wikimedia.org/wikipedia/en/thumb/1/1b/Semi-protection-shackle.svg/20px-Semi-protection-shackle.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/en/thumb/1/1b/Semi-protection-shackle.svg/30px-Semi-protection-shackle.svg.png 1.5x, //upload.wikimedia.org/wikipedia/en/thumb/1/1b/Semi-protection-shackle.svg/40px-Semi-protection-shackle.svg.png 2x\" width=\"20\"/></a>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# find the first `a` tag that has an `href` attribute.\n",
    "link = soup.find('a', href=True)\n",
    "\n",
    "display(link)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section Two: Tag Objects\n",
    "\n",
    "It's not apparent at first, but any object that we look for is returned to us as another object that has specific properties about it. To be more explicit, it's a beautiful soup tag object. This particular object has particular features that make retrieving different portions of it more manageable.\n",
    "\n",
    ">- **name:**  This is the name of the tag.\n",
    ">- **text:**  This is the readable text of the label if there is any.\n",
    ">- **attrs:** This returns a dictionary of each attribute behaving as the key and the corresponding info as the value.\n",
    "\n",
    "To access any of the attributes, we call the object and then pass through the key which will return the value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.element.Tag"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'a'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'href': '/wiki/Wikipedia:Protection_policy#semi',\n",
       " 'title': 'This article is semi-protected until June 21, 2019 at 18:17 UTC.'}"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'/wiki/Wikipedia:Protection_policy#semi'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'This article is semi-protected until June 21, 2019 at 18:17 UTC.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# what is the type.\n",
    "display(type(link))\n",
    "\n",
    "# define the name.\n",
    "display(link.name)\n",
    "\n",
    "# get the dictionary attribute.\n",
    "display(link.attrs)\n",
    "\n",
    "# lets get each value in that dictionary, luckily there is only 2.\n",
    "display(link['href'])\n",
    "display(link['title'])\n",
    "\n",
    "# we can also get the text, if there is any. In this case there wasn't any text to return so it just returns an empty string.\n",
    "display(link.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section Three: Finding Tags & Content\n",
    "\n",
    "We've seen how to search for one tag, but what if want to search for multiple tags of the same type. Well, this is almost identical to searching for one tag we change the method we use to **`find_all`**. In this example, I use `find_all` to find all the `a` tags in the document, I still only want the ones that have a link, so I keep the `href` attribute set to the value `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.element.ResultSet"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/wiki/Wikipedia:Protection_policy#semi\n",
      "#mw-head\n",
      "#p-search\n",
      "/wiki/Great_Patriotic_War_(term)\n",
      "/wiki/The_Great_Patriotic_War_(The_Americans)\n",
      "/wiki/Patriotic_War_of_1812\n",
      "/wiki/European_theatre_of_World_War_II\n",
      "/wiki/World_War_II\n",
      "/wiki/File:EasternFrontWWIIcolage.png\n",
      "/wiki/Ilyushin_Il-2\n"
     ]
    }
   ],
   "source": [
    "# find all the links in the document.\n",
    "links = soup.find_all('a', href=True)\n",
    "\n",
    "# what is the type of this links object, it should be a bs4.element.ResultSet.\n",
    "display(type(links))\n",
    "\n",
    "# we can iterate through this result set using a for loop. In this loop I grab the href and limit the result sets to 10 items.\n",
    "for link in links[0:10]:\n",
    "    print(link['href'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've seen how using `find_all` can make the process of getting the tags we want significantly easier. However, we just looked at some simple examples. This particular method is very flexible as it allows us to find some of the following:\n",
    "\n",
    ">- Multiple Tags\n",
    ">- Tags with specific attributes\n",
    ">- Tags with specific attributes & attribute values.\n",
    ">- Using functions to narrow our search.\n",
    "\n",
    "Here are some more \"advanced\" examples of using `find_all`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# return a list of all the <table> & and <a> tag \n",
    "tables_and_links = soup.find_all(['a', 'table'])\n",
    "\n",
    "# return a list of all the <table> tag with a 'content' attribute\n",
    "tables_with_cont = soup.find_all('th', style=\"width: 10%;\")\n",
    "\n",
    "# return a list of all the <div> tags that have a class attribute of 'navbox'\n",
    "div_with_nav = soup.find_all('div','navbox')\n",
    "\n",
    "# define a function to find the item\n",
    "def list_with_links(tag):\n",
    "    return tag.name == 'li' and len(tag.find_all('a'))> 7\n",
    "\n",
    "# use the function with find_all\n",
    "list_items_with_links = soup.find_all(list_with_links)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section Four: The Beautiful Soup Family Tree\n",
    "\n",
    "Navigating the HTML tree is pretty simple, once you understand it's relationship to other elements in the document. For example, multiple items can fall under the same tag. Take, for example, a simple document provided by BeautifulSoup\n",
    "\n",
    "`<html>\n",
    "  <body>\n",
    "   <a>\n",
    "    <b>\n",
    "     text1\n",
    "    </b>\n",
    "    <c>\n",
    "     text2\n",
    "    </c>\n",
    "   </a>\n",
    "  </body>\n",
    " </html>`\n",
    " \n",
    "Let's explore the different types of relationships available to us in the BeautifulSoup library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      " <body>\n",
      "  <a>\n",
      "   <b>\n",
      "    text1\n",
      "   </b>\n",
      "   <c>\n",
      "    text2\n",
      "   </c>\n",
      "  </a>\n",
      " </body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "# define the simple tree we see above.\n",
    "simple_tree = \"\"\"<html><body><a><b>text1</b><c>text2</c></a></body></html>\"\"\"\n",
    "\n",
    "# pass the simple tree into our Parser to create some simple soup.\n",
    "simple_soup = BeautifulSoup(simple_tree, 'html.parser')\n",
    "\n",
    "# we can always print it in a familiar structure.\n",
    "print(simple_soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Going down the tree: Children\n",
    "We notice that in our HTML code above that we have tags within tags, the tags contained in other tags are the **Children** and in Beautiful Soup we have different ways for moving through these children. If you look at the simple tree above, both tag `b` and tag `c` are children of the tag `a`. Let's see how we can access those children."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<b>text1</b>, <c>text2</c>]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<c>text2</c>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# if we want just a list of the children we could use the Contents attribute\n",
    "a_content = simple_soup.a.contents\n",
    "\n",
    "# display the list\n",
    "display(a_content)\n",
    "\n",
    "# display the first child\n",
    "display(a_content[0])\n",
    "\n",
    "# if we want to create a generator to iterate over them vice a list we can use the `children` attribute\n",
    "for child in simple_soup.a.children:\n",
    "    display(child)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Going down the tree: Descendants\n",
    "When using the `contents` & `childrens` attribute, we are only considering tags direct children, the ones that fall underneath that tag. However, if wanted all the tags beneath the children we would need to use the `descendants` attribute. The `descendants` attribute returns not just the children, but the children within the children."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<b>text1</b>\n",
      "text1\n",
      "<c>text2</c>\n",
      "text2\n"
     ]
    }
   ],
   "source": [
    "# let's first verify there is a difference\n",
    "display(len(list(simple_soup.a.children)))\n",
    "display(len(list(simple_soup.a.descendants)))\n",
    "\n",
    "# greate so we know there is a difference, lets iterate through the descendants\n",
    "for descendant in simple_soup.a.descendants:\n",
    "    print(descendant)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Going up the tree: Parent & Parents\n",
    "We now see that we can access the tags that are direct descendants of the tag in question. We refer to the accessing of these tags as going **down the tree**. If we can go down the tree, we should be able to go up the tree right? Correct, this is possible using the `parent` & `parents` attribute. The parent is **the tag that contains the tag we are referring to**. Here is how we access the parent/parents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<body><a><b>text1</b><c>text2</c></a></body>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<html><body><a><b>text1</b><c>text2</c></a></body></html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<body><a><b>text1</b><c>text2</c></a></body>\n",
      "<html><body><a><b>text1</b><c>text2</c></a></body></html>\n",
      "<html><body><a><b>text1</b><c>text2</c></a></body></html>\n"
     ]
    }
   ],
   "source": [
    "# lets see if tag `a` has a parent\n",
    "display(simple_soup.a.parent)\n",
    "\n",
    "# is there a parent of the parent?\n",
    "display(simple_soup.a.parent.parent)\n",
    "\n",
    "# looks like there is, but we can write this more concisely using a generator along with a for loop\n",
    "for parent in simple_soup.a.parents:\n",
    "    print(parent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Going Sideways\n",
    "In the simple tree up above, we would consider that element `<b>` and element `<c>` are **siblings** because they both fall under the same `a` tag element **the parent**. The `a` tag signifies that means we can navigate between the two using the `next_sibling` & `previous_sibling` properties. When we go from one sibling to the next, we are said to be going sideways because they both exist under the same tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<c>text2</c>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "None"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# grab the b tag\n",
    "display(simple_soup.b)\n",
    "\n",
    "# grab the next sibling, this should be the c tag.\n",
    "display(simple_soup.b.next_sibling)\n",
    "\n",
    "# grab the previous sibling, this should be the b tag.\n",
    "display(simple_soup.c.previous_sibling)\n",
    "\n",
    "# if I call 'previous_sibling' on the b tag it will return none, that's because the 'b' tag has no siblings before it\n",
    "# grab the next sibling, this should be the c tag.\n",
    "display(simple_soup.b.previous_sibling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<div class=\"hatnote navigation-not-searchable\" role=\"note\">\"Great Patriotic War\" redirects here. For a discussion of the term itself, see <a href=\"/wiki/Great_Patriotic_War_(term)\" title=\"Great Patriotic War (term)\">Great Patriotic War (term)</a>. For the episode of The Americans, see <a href=\"/wiki/The_Great_Patriotic_War_(The_Americans)\" title=\"The Great Patriotic War (The Americans)\">The Great Patriotic War (The Americans)</a>.</div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'\\n'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<p class=\"mw-empty-elt\">\n",
       "</p>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# grab the first paragraph, and then get it's sibling\n",
    "display(soup.p.next_sibling)\n",
    "display(soup.p.next_sibling.next_sibling)\n",
    "\n",
    "# define the bottom\n",
    "bottom = soup.p.next_sibling.next_sibling\n",
    "\n",
    "# grab the first paragraph, and then get it's sibling\n",
    "display(bottom.previous_sibling)\n",
    "display(bottom.previous_sibling.previous_sibling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Going back & forth\n",
    "One of the simplest ways to navigate our simple tree is to use the `next_element` and the `previous_element` properties. All these two properties do is proceed down from top to bottom. If I want to go forward down my tree, I will use `next_element` and if I wanted to go backward up my tree I would specify the `previous_element`. Keep in mind that you don't have started at the beginning of the code to start traversing it, you can call the tag you want to start at and then go down."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Tip:</b> Now a lot of people ask this, \"What's the difference between the next element versus the next sibling?\". The answer is simple when we talk about the next element we mean the next item that Beautiful Soup parsed. Just because it was the next item parsed does not mean it's a sibling.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<body><a><b>text1</b><c>text2</c></a></body>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<a><b>text1</b><c>text2</c></a>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# grab the body\n",
    "display(simple_soup.body)\n",
    "\n",
    "# grab the next element after the body, which would be the 'a' tag.\n",
    "display(simple_soup.body.next_element)\n",
    "\n",
    "# keep going forward\n",
    "display(simple_soup.body.next_element.next_element)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<a><b>text1</b><c>text2</c></a>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<body><a><b>text1</b><c>text2</c></a></body>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# grab the b tag\n",
    "display(simple_soup.b)\n",
    "\n",
    "# grab the PREVIOUS element after the body, which would be the 'a' tag.\n",
    "display(simple_soup.b.previous_element)\n",
    "\n",
    "# keep going backwards\n",
    "display(simple_soup.b.previous_element.previous_element)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<caption>Comparative strengths of combat forces, Eastern Front, 1941–1945<sup class=\"reference\" id=\"cite_ref-37\"><a href=\"#cite_note-37\">[37]</a></sup><sup class=\"reference\" id=\"cite_ref-FOOTNOTEGlantz1998107_38-0\"><a href=\"#cite_note-FOOTNOTEGlantz1998107-38\">[38]</a></sup><sup class=\"reference\" id=\"cite_ref-FOOTNOTEGlantzHouse199568_39-0\"><a href=\"#cite_note-FOOTNOTEGlantzHouse199568-39\">[39]</a></sup>\n",
       "</caption>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'\\n'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "', leader of the Free French, who thought that it was important for French servicemen to serve on all fronts.\\n'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's do an example with one of our tables\n",
    "my_table = soup.find_all('table')[5]\n",
    "\n",
    "# using next element\n",
    "display(my_table.next_element)\n",
    "display(my_table.next_element.next_element)\n",
    "\n",
    "# using previous element\n",
    "display(my_table.previous_element)\n",
    "display(my_table.previous_element.previous_element)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Sibling & Element Generators\n",
    "What if I want all the siblings or all the next/previous elements that belong to a particular tag? Well up above, I made the process a little more complicated then it had to. What we could've done up above is instead of writing `next_sibling.next_sibling` we could've just called the `next_siblings` iterator to simple fetch all the siblings of that particular tag, and then we could iterate through each one of them. Here is how the above examples would look if we chose to follow that path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<body><a><b>text1</b><c>text2</c></a></body>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<generator object next_elements at 0x0000020724935DB0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<a><b>text1</b><c>text2</c></a>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'text1'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<c>text2</c>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'text2'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# grab the body\n",
    "display(simple_soup.body)\n",
    "\n",
    "# create a next_elements generator\n",
    "display(simple_soup.body.next_elements)\n",
    "\n",
    "# loop through each element in the generator.\n",
    "for element in simple_soup.body.next_elements:    \n",
    "    display(element)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<generator object previous_elements at 0x0000020726751DB0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<a><b>text1</b><c>text2</c></a>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<body><a><b>text1</b><c>text2</c></a></body>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<html><body><a><b>text1</b><c>text2</c></a></body></html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# grab the b tag\n",
    "display(simple_soup.b)\n",
    "\n",
    "# create a previous_elements generator\n",
    "display(simple_soup.b.previous_elements)\n",
    "\n",
    "# loop through each element in the generator.\n",
    "for element in simple_soup.b.previous_elements:    \n",
    "    display(element)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Find_Parent & Find_Parents\n",
    "Okay, instead of navigating through the tags based on their relationship to other tags. What if we wanted to find a particular tags family member? Is this possible? It is! The first example we will look is using `find_parent` & `find_parents`, both of these will find a particular tags parent or parents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a><b>text1</b><c>text2</c></a>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[<a><b>text1</b><c>text2</c></a>,\n",
       " <body><a><b>text1</b><c>text2</c></a></body>,\n",
       " <html><body><a><b>text1</b><c>text2</c></a></body></html>,\n",
       " <html><body><a><b>text1</b><c>text2</c></a></body></html>]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "simple_soup.b\n",
    "\n",
    "display(simple_soup.b.find_parent())\n",
    "\n",
    "display(simple_soup.b.find_parents())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Finding Siblings & elements\n",
    "If we can search for parents, can we search for siblings and elements related to a tag? Is it possible to find the previous/next elements or tags for a given tag? The answer is yes, the examples below are identical to the `find_parent` & `find_parents` but now we are dealing with both `element` & `siblings`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<c>text2</c>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[<c>text2</c>]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "simple_soup.b\n",
    "\n",
    "display(simple_soup.b.find_next_sibling())\n",
    "\n",
    "display(simple_soup.b.find_next_siblings())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[<b>text1</b>]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "simple_soup.c\n",
    "\n",
    "display(simple_soup.c.find_previous_sibling())\n",
    "\n",
    "display(simple_soup.c.find_previous_siblings())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[<b>text1</b>, <c>text2</c>]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "simple_soup.a\n",
    "\n",
    "display(simple_soup.a.find_next())\n",
    "\n",
    "display(simple_soup.a.find_all_next())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[<b>text1</b>,\n",
       " <a><b>text1</b><c>text2</c></a>,\n",
       " <body><a><b>text1</b><c>text2</c></a></body>,\n",
       " <html><body><a><b>text1</b><c>text2</c></a></body></html>]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "simple_soup.c\n",
    "\n",
    "display(simple_soup.c.find_previous())\n",
    "\n",
    "display(simple_soup.c.find_all_previous())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Section Four Extracting & Manipulating Text\n",
    "In the previous examples, we saw how to navigate and find tags that we are searching for. Assuming we found the tag we want, what are some other ways we can manipulate them? In the following examples, we will explore how to manage tags to get the information we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<table class=\"wikitable\">\n",
      " <caption>\n",
      "  Comparative strengths of combat forces, Eastern Front, 1941–1945\n",
      "  <sup class=\"reference\" id=\"cite_ref-37\">\n",
      "   <a href=\"#cite_note-37\">\n",
      "    [37]\n",
      "   </a>\n",
      "  </sup>\n",
      "  <sup class=\"reference\" id=\"cite_ref-FOOTNOTEGlantz1998107_38-0\">\n",
      "   <a href=\"#cite_note-FOOTNOTEGlantz1998107-38\">\n",
      "    [38]\n",
      "   </a>\n",
      "  </sup>\n",
      "  <sup class=\"reference\" id=\"cite_ref-FOOTNOTEGlantzHouse199568_39-0\">\n",
      "   <a href=\"#cite_note-FOOTNOTEGlantzHouse199568-39\">\n",
      "    [39]\n",
      "   </a>\n",
      "  </sup>\n",
      " </caption>\n",
      " <tbody>\n",
      "  <tr>\n",
      "   <th style=\"my new width\">\n",
      "    Date\n",
      "   </th>\n",
      "   <th style=\"width: 45%;\">\n",
      "    Axis forces\n",
      "   </th>\n",
      "   <th style=\"width: 45%;\">\n",
      "    Soviet forces\n",
      "   </th>\n",
      "  </tr>\n",
      "  <tr>\n",
      "   <td>\n",
      "    June 1941\n",
      "   </td>\n",
      "   <td>\n",
      "    3,050,000 Germans, 67,000 (northern Norway); 500,000 Finns, 150,000 Romanians,\n",
      "    <a href=\"/wiki/Italian_Expeditionary_Corps_in_Russia\" title=\"Italian Expeditionary Corps in Russia\">\n",
      "     62,000 Italians\n",
      "    </a>\n",
      "    <br/>\n",
      "    Total:\n",
      "    <b>\n",
      "     3,829,000\n",
      "    </b>\n",
      "    in the east (80% of the German Army)\n",
      "   </td>\n",
      "   <td>\n",
      "    2,680,000 active in Western Military Districts out of\n",
      "    <b>\n",
      "     5,500,000\n",
      "    </b>\n",
      "    (overall); 14,000,000 mobilizable reserves (former conscripts who fulfilled their mandatory service prior)\n",
      "   </td>\n",
      "  </tr>\n",
      "  <tr>\n",
      "   <td>\n",
      "    June 1942\n",
      "   </td>\n",
      "   <td>\n",
      "    2,600,000 Germans, 90,000 (northern Norway); 430,000 Finns, 600,000 Romanians, Hungarians, and Italians\n",
      "    <br/>\n",
      "    Total:\n",
      "    <b>\n",
      "     3,720,000\n",
      "    </b>\n",
      "    in the east (80% of the German Army)\n",
      "   </td>\n",
      "   <td>\n",
      "    5,313,000 (front); 383,000 (hospital)\n",
      "    <br/>\n",
      "    Total:\n",
      "    <b>\n",
      "     9,350,000\n",
      "    </b>\n",
      "   </td>\n",
      "  </tr>\n",
      "  <tr>\n",
      "   <td>\n",
      "    July 1943\n",
      "   </td>\n",
      "   <td>\n",
      "    3,403,000 Germans, 80,000 (northern Norway); 400,000 Finns, 150,000 Romanians and Hungarians\n",
      "    <br/>\n",
      "    Total:\n",
      "    <b>\n",
      "     3,933,000\n",
      "    </b>\n",
      "    in the east (63% of the German Army)\n",
      "   </td>\n",
      "   <td>\n",
      "    6,724,000 (front); 446,445 (hospital);\n",
      "    <br/>\n",
      "    Total:\n",
      "    <b>\n",
      "     10,300,000\n",
      "    </b>\n",
      "   </td>\n",
      "  </tr>\n",
      "  <tr>\n",
      "   <td>\n",
      "    June 1944\n",
      "   </td>\n",
      "   <td>\n",
      "    2,460,000 Germans, 60,000 (northern Norway); 300,000 Finns, 550,000 Romanians and Hungarians\n",
      "    <br/>\n",
      "    Total:\n",
      "    <b>\n",
      "     3,370,000\n",
      "    </b>\n",
      "    in the east (62% of the German Army)\n",
      "   </td>\n",
      "   <td>\n",
      "    <b>\n",
      "     6,425,000\n",
      "    </b>\n",
      "   </td>\n",
      "  </tr>\n",
      "  <tr>\n",
      "   <td>\n",
      "    Jan. 1945\n",
      "   </td>\n",
      "   <td>\n",
      "    2,230,000 Germans, 100,000 Hungarians\n",
      "    <br/>\n",
      "    Total:\n",
      "    <b>\n",
      "     2,330,000\n",
      "    </b>\n",
      "    in the east (60% of the German Army)\n",
      "   </td>\n",
      "   <td>\n",
      "    <b>\n",
      "     6,532,000\n",
      "    </b>\n",
      "    (360,000 Poles, Romanians, Bulgarians, and Czechs)\n",
      "   </td>\n",
      "  </tr>\n",
      "  <tr>\n",
      "   <td>\n",
      "    Apr. 1945\n",
      "   </td>\n",
      "   <td>\n",
      "    <b>\n",
      "     1,960,000\n",
      "    </b>\n",
      "   </td>\n",
      "   <td>\n",
      "    <b>\n",
      "     6,410,000\n",
      "    </b>\n",
      "    (450,000 Poles, Romanians, Bulgarians, and Czechs)\n",
      "   </td>\n",
      "  </tr>\n",
      " </tbody>\n",
      "</table>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# get my table\n",
    "table = soup.find_all('table')[5]\n",
    "print(table.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing Attribute Values\n",
    "Maybe you're not happy with the current value of an attribute in the HTML code, well lucky for you we can change the attribute value using beautiful soup. In the example, we call the attribute in question and then assign it a new value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'#cite_note-37'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<a href=\"#cite_note-37\" style=\"my new width\">[37]</a>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# grab a header tag from the table\n",
    "header_tag = table.a\n",
    "\n",
    "# display the style attribute content\n",
    "display(header_tag['href'])\n",
    "\n",
    "# change the style attribute content\n",
    "header_tag['style'] = 'my new width'\n",
    "\n",
    "# display the changes\n",
    "header_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same goes for the string, if we want to add a string to it we can."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a href=\"#cite_note-37\" style=\"my new width\">My New String</a>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add a string to a tag\n",
    "table.a.string = 'My New String'\n",
    "\n",
    "# display the tag with a new string now.\n",
    "table.a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deleting Attribute Values\n",
    "The alternative to adding or changing a value is deleting it. Well again, BeautifulSoup offers a mechanism where we can easily remove tags and content from our soup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a href=\"#cite_note-37\" style=\"my new width\">My New String</a>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<a href=\"#cite_note-37\" style=\"my new width\"></a>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#removes the content of a tag\n",
    "tag = table.a\n",
    "\n",
    "display(tag)\n",
    "\n",
    "tag.clear()\n",
    "\n",
    "tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting Tags\n",
    "Sometimes we want to work with a tag individually and not associated with any of its descendants or parents. If we use the `extract()` method, we yank that tag out of the HTML code and assign it to a new variable. Keep in mind once it's yanked, it's yanked. If you want the first soup to contain it again, you'll have to run the full code again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(table.tbody)\n",
    "\n",
    "th_tag = table.tbody.th.extract()\n",
    "\n",
    "display(th_tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting Strings\n",
    "Many times I would say most, we are scraping a website for particular text. Make the process of getting the text accessible and quick by leveraging the built-in `string` & `strings` attributes which return the first or all strings in the document, respectively. Remember that if you use the `strings` attribute, it produces a generator which you can loop through. Also, if you want all the strings minus the `t\\` & `n\\` use the `stripped_strings` attribute instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Axis forces\\n'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Axis forces\n",
      "\n",
      "\n",
      "\n",
      "Soviet forces\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "June 1941\n",
      "\n",
      "\n",
      "3,050,000 Germans, 67,000 (northern Norway); 500,000 Finns, 150,000 Romanians, \n",
      "-----------------------------------------------\n",
      "Axis forces\n",
      "Soviet forces\n",
      "June 1941\n",
      "3,050,000 Germans, 67,000 (northern Norway); 500,000 Finns, 150,000 Romanians,\n",
      "62,000 Italians\n",
      "Total:\n",
      "3,829,000\n",
      "in the east (80% of the German Army)\n",
      "2,680,000 active in Western Military Districts out of\n",
      "5,500,000\n"
     ]
    }
   ],
   "source": [
    "# grab a single string that belongs to the table header\n",
    "display(table.th.string)\n",
    "\n",
    "# get all the strings from the table body using the strings attribute.\n",
    "for string in list(table.tbody.strings)[0:10]:\n",
    "    print(string)\n",
    "\n",
    "print('-----------------------------------------------')\n",
    "\n",
    "# get all the strings from the table body using the strings attribute.\n",
    "for string in list(table.tbody.stripped_strings)[0:10]:\n",
    "    print(string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More Wikipedia Examples\n",
    "Okay, so now that we know, the whole structure of our tree let's put it to work. In the example below I will find all the tables in my soup, specify the sixth one because I like that one, and then create all of my necessary generators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eastern Front\n",
      "Part of the\n",
      "European theatre\n",
      "of\n",
      "World War II\n",
      "Clockwise from top left\n",
      ": Soviet\n",
      "Il-2\n",
      "ground attack aircraft in Berlin sky; German\n",
      "Tiger I\n",
      "Date\n",
      "22 June 1941\n",
      "(\n",
      "1941-06-22\n",
      ")\n",
      "– 25 May 1945\n",
      "(\n",
      "1945-05-25\n",
      ")\n",
      "[2]\n",
      "v\n",
      "t\n",
      "e\n",
      "Campaigns of\n",
      "World War II\n",
      "Europe\n",
      "Poland\n",
      "Phoney War\n",
      "Winter War\n",
      "Denmark & Norway\n",
      "v\n",
      "t\n",
      "e\n",
      "Eastern Front\n",
      "Naval warfare\n",
      "Baltic Sea\n",
      "Black Sea\n",
      "Arctic Ocean\n",
      "1941\n",
      "Barbarossa\n",
      "Part of\n",
      "a series\n",
      "on the\n",
      "History of the\n",
      "Union of Soviet\n",
      "Socialist Republics\n",
      "1917–1927\n",
      "Revolutionary beginnings\n",
      "Revolution\n",
      "Civil War\n",
      "Date\n",
      "Axis forces\n",
      "Soviet forces\n",
      "June 1941\n",
      "3,050,000 Germans, 67,000 (northern Norway); 500,000 Finns, 150,000 Romanians,\n",
      "62,000 Italians\n",
      "Total:\n",
      "3,829,000\n",
      "in the east (80% of the German Army)\n",
      "2,680,000 active in Western Military Districts out of\n",
      "Year\n",
      "Amount\n",
      "(tons)\n",
      "%\n",
      "1941\n",
      "360,778\n",
      "2.1\n",
      "1942\n",
      "2,453,097\n",
      "14\n",
      "This section\n",
      "does not\n",
      "cite\n",
      "any\n",
      "sources\n",
      ".\n",
      "Please help\n",
      "improve this section\n",
      "by\n",
      "adding citations to reliable sources\n",
      "This section\n",
      "does not\n",
      "cite\n",
      "any\n",
      "sources\n",
      ".\n",
      "Please help\n",
      "improve this section\n",
      "by\n",
      "adding citations to reliable sources\n",
      "This section\n",
      "needs additional citations for\n",
      "verification\n",
      ".\n",
      "Please help\n",
      "improve this article\n",
      "by\n",
      "adding citations to reliable sources\n",
      ". Unsourced material may be challenged and removed.\n",
      "Find sources:\n",
      "This section\n",
      "does not\n",
      "cite\n",
      "any\n",
      "sources\n",
      ".\n",
      "Please help\n",
      "improve this section\n",
      "by\n",
      "adding citations to reliable sources\n",
      "This section\n",
      "needs additional citations for\n",
      "verification\n",
      ".\n",
      "Please help\n",
      "improve this article\n",
      "by\n",
      "adding citations to reliable sources\n",
      ". Unsourced material may be challenged and removed.\n",
      "Find sources:\n",
      "Year\n",
      "Coal\n",
      "(million tonnes, Germany includes lignite and bituminous types)\n",
      "Steel\n",
      "(million tonnes)\n",
      "Aluminium\n",
      "(thousand tonnes)\n",
      "Oil\n",
      "(million tonnes)\n",
      "German\n",
      "Year\n",
      "Tanks and self-\n",
      "propelled guns\n",
      "Soviet\n",
      "German\n",
      "Italian\n",
      "Hungarian\n",
      "Romanian\n",
      "Japanese\n",
      "1941\n",
      "Year\n",
      "Aircraft\n",
      "Soviet\n",
      "German\n",
      "Italian\n",
      "Hungarian\n",
      "Romanian\n",
      "Japanese\n",
      "1941\n",
      "15,735\n",
      "Year\n",
      "Industrial Labour\n",
      "Foreign Labour\n",
      "Total Labour\n",
      "Soviet\n",
      "German\n",
      "Soviet\n",
      "German\n",
      "Total Soviet\n",
      "Total German\n",
      "Forces fighting with the Axis\n",
      "Total Dead\n",
      "KIA\n",
      "/\n",
      "DOW\n",
      "/\n",
      "MIA\n",
      "Prisoners taken by the Soviets\n",
      "Prisoners who died in Captivity\n",
      "WIA\n",
      "Forces fighting with the Soviet Union\n",
      "Total Dead\n",
      "KIA/DOW/MIA\n",
      "Prisoners taken by the Axis\n",
      "Prisoners who died in captivity\n",
      "WIA (not including DOW)\n",
      "Soviet\n",
      "8,668,400–10,000,000\n",
      "6,829,600\n",
      "4,059,000 (military personnel only)–5,700,000\n",
      "Wikisource\n",
      "has original text related to this article:\n",
      "Adolf Hitler's Letter to Benito Mussolini Explaining the Invasion of the Soviet Union\n",
      "Wikisource\n",
      "has original text related to this article:\n",
      "The Führer to the German People: 22 June 1941\n",
      "Wikisource\n",
      "has original text related to this article:\n",
      "Adolf Hitler's Order of the Day to the German Troops on the Eastern Front (2 October 1941)\n",
      "Wikisource\n",
      "has original text related to this article:\n",
      "Adolf Hitler Explains His Reasons for Invading the Soviet Union\n",
      "Wikisource\n",
      "has original text related to this article:\n",
      "Adolf Hitler's Proclamation to the German Army on His Assumption of Direct Command\n",
      "Wikisource\n",
      "has original text related to this article:\n",
      "Adolf Hitler's Speech at the Berlin Sportpalast (30 January 1942)\n",
      "Portals\n",
      "Access related topics\n",
      "Hungary portal\n",
      "Military of Germany portal\n",
      "Poland portal\n",
      "Romania portal\n",
      "Soviet Union portal\n",
      "World War II portal\n",
      "Find out more on Wikipedia's\n",
      "Sister projects\n",
      "v\n",
      "t\n",
      "e\n",
      "World War II\n",
      "Outline\n",
      "Military engagements\n",
      "Battles\n",
      "Operations\n",
      "Commanders\n",
      "Casualties\n",
      "Allies\n",
      ".mw-parser-output .nobold{font-weight:normal}\n",
      "(\n",
      "leaders\n",
      ")\n",
      "Australia\n",
      "Belgium\n",
      "Brazil\n",
      "Canada\n",
      "China\n",
      "Prelude\n",
      "Africa\n",
      "Asia\n",
      "Europe\n",
      "1939\n",
      "Poland\n",
      "Phoney War\n",
      "Winter War\n",
      "Atlantic\n",
      "Changsha\n",
      "General\n",
      "Famines\n",
      "Bengal famine of 1943\n",
      "Chinese famine of 1942–43\n",
      "Greek Famine of 1941-1944\n",
      "Dutch famine of 1944–45\n",
      "Vietnamese Famine of 1945\n",
      "Air warfare of World War II\n",
      "In Europe\n",
      "Blitzkrieg\n",
      "Famines\n",
      "Bengal famine of 1943\n",
      "Chinese famine of 1942–43\n",
      "Greek Famine of 1941-1944\n",
      "Dutch famine of 1944–45\n",
      "Vietnamese Famine of 1945\n",
      "Air warfare of World War II\n",
      "In Europe\n",
      "Blitzkrieg\n",
      "Comparative military ranks\n",
      "v\n",
      "t\n",
      "e\n",
      "Wehrmacht\n",
      "Army Group Rear Areas\n",
      "during the\n",
      "German–Soviet War\n",
      ", 1941–45\n",
      "Army Group Rear Area\n",
      "North\n",
      "v\n",
      "t\n",
      "e\n",
      "Annual\n",
      "Moscow military parade on Victory Day in the Great Patriotic War 1941-1945\n",
      "By year\n",
      "1945\n",
      "1965\n",
      "1985\n",
      "1990\n",
      "v\n",
      "t\n",
      "e\n",
      "Armed conflicts involving\n",
      "Russia\n",
      "(incl.\n",
      "Imperial\n",
      "and\n",
      "Soviet\n",
      "times)\n"
     ]
    }
   ],
   "source": [
    "# grab all the tables\n",
    "tables = soup.find_all('table')\n",
    "\n",
    "# loop through each table\n",
    "for table in tables:    \n",
    "    \n",
    "    # get all the strings from the table body using the strings attribute.\n",
    "    for string in list(table.tbody.stripped_strings)[0:10]:\n",
    "        print(string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get my table\n",
    "table = soup.find_all('tr')[5]\n",
    "\n",
    "# define my generators\n",
    "the_parents = table.parents\n",
    "the_children = table.children\n",
    "the_descendants = table.descendants\n",
    "\n",
    "next_sibs = table.next_siblings\n",
    "prev_sibs = table.previous_siblings\n",
    "next_elem = table.next_elements\n",
    "prev_elem = table.previous_elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------\n",
      "<th style=\"padding-right:1em\">Location</th>\n",
      "-----------------------------\n",
      "<td><div class=\"location\">Europe east of Germany: <a href=\"/wiki/Central_and_Eastern_Europe\" title=\"Central and Eastern Europe\">Central and Eastern Europe</a>, in later stages Germany and Austria</div></td>\n"
     ]
    }
   ],
   "source": [
    "for child in list(the_children):\n",
    "    print('-----------------------------')\n",
    "    print(child)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------\n",
      "<th style=\"padding-right:1em\">Location</th>\n",
      "-----------------------------\n",
      "Location\n",
      "-----------------------------\n",
      "<td><div class=\"location\">Europe east of Germany: <a href=\"/wiki/Central_and_Eastern_Europe\" title=\"Central and Eastern Europe\">Central and Eastern Europe</a>, in later stages Germany and Austria</div></td>\n",
      "-----------------------------\n",
      "<div class=\"location\">Europe east of Germany: <a href=\"/wiki/Central_and_Eastern_Europe\" title=\"Central and Eastern Europe\">Central and Eastern Europe</a>, in later stages Germany and Austria</div>\n",
      "-----------------------------\n",
      "Europe east of Germany: \n",
      "-----------------------------\n",
      "<a href=\"/wiki/Central_and_Eastern_Europe\" title=\"Central and Eastern Europe\">Central and Eastern Europe</a>\n",
      "-----------------------------\n",
      "Central and Eastern Europe\n",
      "-----------------------------\n",
      ", in later stages Germany and Austria\n"
     ]
    }
   ],
   "source": [
    "for descendant in the_descendants:  \n",
    "    print('-----------------------------')\n",
    "    print(descendant)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------\n",
      "<tr><th style=\"padding-right:1em\">Result</th><td>\n",
      "<p>Soviet victory\n",
      "</p>\n",
      "<ul><li>Fall of the <a class=\"mw-redirect\" href=\"/wiki/Third_Reich\" title=\"Third Reich\">Third Reich</a></li>\n",
      "<li><a class=\"mw-redirect\" href=\"/wiki/Allied_occupation_of_Germany\" title=\"Allied occupation of Germany\">Allied occupation of Germany</a></li>\n",
      "<li>Beginning of the <a href=\"/wiki/Cold_War\" title=\"Cold War\">Cold War</a> and creation of the <a href=\"/wiki/Eastern_Bloc\" title=\"Eastern Bloc\">Eastern Bloc</a> and the <a href=\"/wiki/Iron_Curtain\" title=\"Iron Curtain\">Iron Curtain</a></li>\n",
      "<li>Beginning of the <a href=\"/wiki/Greek_Civil_War\" title=\"Greek Civil War\">Greek Civil War</a></li></ul></td></tr>\n",
      "----------------------------\n",
      "<tr><th style=\"padding-right:1em\">Territorial<br/>changes</th><td>\n",
      "<li>The Soviet Union occupies <a href=\"/wiki/Central_Europe\" title=\"Central Europe\">Central</a>, <a href=\"/wiki/Eastern_Europe\" title=\"Eastern Europe\">Eastern</a>, <a class=\"mw-redirect\" href=\"/wiki/Northeastern_Europe\" title=\"Northeastern Europe\">Northeastern</a> and <a class=\"mw-redirect\" href=\"/wiki/Southeastern_Europe\" title=\"Southeastern Europe\">Southeastern Europe</a> and establishes pro-Soviet <a href=\"/wiki/Communist_state\" title=\"Communist state\">communist</a> <a href=\"/wiki/Puppet_state\" title=\"Puppet state\">puppet governments</a> in countries including <a href=\"/wiki/People%27s_Republic_of_Bulgaria\" title=\"People's Republic of Bulgaria\">Bulgaria</a>, <a href=\"/wiki/Czechoslovak_Socialist_Republic\" title=\"Czechoslovak Socialist Republic\">Czechoslovakia</a>, <a href=\"/wiki/Hungarian_People%27s_Republic\" title=\"Hungarian People's Republic\">Hungary</a>, <a href=\"/wiki/Polish_People%27s_Republic\" title=\"Polish People's Republic\">Poland</a>, <a class=\"mw-redirect\" href=\"/wiki/Romanian_People%27s_Republic\" title=\"Romanian People's Republic\">Romania</a>, and <a href=\"/wiki/East_Germany\" title=\"East Germany\">Eastern Germany</a>.</li>\n",
      "<li>Establishment of the <a class=\"mw-redirect\" href=\"/wiki/Federal_People%27s_Republic_of_Yugoslavia\" title=\"Federal People's Republic of Yugoslavia\">Federal People's Republic of Yugoslavia</a></li>\n",
      "<li><a href=\"/wiki/History_of_Germany_(1945%E2%80%931990)\" title=\"History of Germany (1945–1990)\">Partition of Germany</a></li>\n",
      "<li><a href=\"/wiki/Territorial_changes_of_Poland_immediately_after_World_War_II\" title=\"Territorial changes of Poland immediately after World War II\">Borders of Poland changed</a>.</li></td></tr>\n"
     ]
    }
   ],
   "source": [
    "for sib in next_sibs:\n",
    "    print('----------------------------')\n",
    "    print(sib)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------\n",
      "<tr><th style=\"padding-right:1em\">Date</th><td>22 June 1941<span style=\"display:none\"> (<span class=\"bday dtstart published updated\">1941-06-22</span>)</span> – 25 May 1945<span style=\"display:none\"> (<span class=\"bday dtstart published updated\">1945-05-25</span>)</span><sup class=\"reference\" id=\"cite_ref-2\"><a href=\"#cite_note-2\">[2]</a></sup><br/>(3 years, 11 months and 3 days)</td></tr>\n"
     ]
    }
   ],
   "source": [
    "for sib in prev_sibs:\n",
    "    print('----------------------------')\n",
    "    print(sib)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------\n",
      "<th style=\"padding-right:1em\">Location</th>\n",
      "----------------------------\n",
      "Location\n"
     ]
    }
   ],
   "source": [
    "# just for speed purposes, let's only do a few next elements\n",
    "next_list = list(next_elem)\n",
    "\n",
    "for elem in next_list[0:2]:\n",
    "    print('----------------------------')\n",
    "    print(elem)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------\n",
      "(3 years, 11 months and 3 days)\n",
      "----------------------------\n",
      "<br/>\n"
     ]
    }
   ],
   "source": [
    "# just for speed purposes, let's only do a few previous elements\n",
    "prev_list = list(prev_elem)\n",
    "\n",
    "for elem in prev_list[0:2]:\n",
    "    print('----------------------------')\n",
    "    print(elem)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------\n",
      "<tbody><tr><th style=\"padding-right:1em\">Date</th><td>22 June 1941<span style=\"display:none\"> (<span class=\"bday dtstart published updated\">1941-06-22</span>)</span> – 25 May 1945<span style=\"display:none\"> (<span class=\"bday dtstart published updated\">1945-05-25</span>)</span><sup class=\"reference\" id=\"cite_ref-2\"><a href=\"#cite_note-2\">[2]</a></sup><br/>(3 years, 11 months and 3 days)</td></tr><tr><th style=\"padding-right:1em\">Location</th><td><div class=\"location\">Europe east of Germany: <a href=\"/wiki/Central_and_Eastern_Europe\" title=\"Central and Eastern Europe\">Central and Eastern Europe</a>, in later stages Germany and Austria</div></td></tr><tr><th style=\"padding-right:1em\">Result</th><td>\n",
      "<p>Soviet victory\n",
      "</p>\n",
      "<ul><li>Fall of the <a class=\"mw-redirect\" href=\"/wiki/Third_Reich\" title=\"Third Reich\">Third Reich</a></li>\n",
      "<li><a class=\"mw-redirect\" href=\"/wiki/Allied_occupation_of_Germany\" title=\"Allied occupation of Germany\">Allied occupation of Germany</a></li>\n",
      "<li>Beginning of the <a href=\"/wiki/Cold_War\" title=\"Cold War\">Cold War</a> and creation of the <a href=\"/wiki/Eastern_Bloc\" title=\"Eastern Bloc\">Eastern Bloc</a> and the <a href=\"/wiki/Iron_Curtain\" title=\"Iron Curtain\">Iron Curtain</a></li>\n",
      "<li>Beginning of the <a href=\"/wiki/Greek_Civil_War\" title=\"Greek Civil War\">Greek Civil War</a></li></ul></td></tr><tr><th style=\"padding-right:1em\">Territorial<br/>changes</th><td>\n",
      "<li>The Soviet Union occupies <a href=\"/wiki/Central_Europe\" title=\"Central Europe\">Central</a>, <a href=\"/wiki/Eastern_Europe\" title=\"Eastern Europe\">Eastern</a>, <a class=\"mw-redirect\" href=\"/wiki/Northeastern_Europe\" title=\"Northeastern Europe\">Northeastern</a> and <a class=\"mw-redirect\" href=\"/wiki/Southeastern_Europe\" title=\"Southeastern Europe\">Southeastern Europe</a> and establishes pro-Soviet <a href=\"/wiki/Communist_state\" title=\"Communist state\">communist</a> <a href=\"/wiki/Puppet_state\" title=\"Puppet state\">puppet governments</a> in countries including <a href=\"/wiki/People%27s_Republic_of_Bulgaria\" title=\"People's Republic of Bulgaria\">Bulgaria</a>, <a href=\"/wiki/Czechoslovak_Socialist_Republic\" title=\"Czechoslovak Socialist Republic\">Czechoslovakia</a>, <a href=\"/wiki/Hungarian_People%27s_Republic\" title=\"Hungarian People's Republic\">Hungary</a>, <a href=\"/wiki/Polish_People%27s_Republic\" title=\"Polish People's Republic\">Poland</a>, <a class=\"mw-redirect\" href=\"/wiki/Romanian_People%27s_Republic\" title=\"Romanian People's Republic\">Romania</a>, and <a href=\"/wiki/East_Germany\" title=\"East Germany\">Eastern Germany</a>.</li>\n",
      "<li>Establishment of the <a class=\"mw-redirect\" href=\"/wiki/Federal_People%27s_Republic_of_Yugoslavia\" title=\"Federal People's Republic of Yugoslavia\">Federal People's Republic of Yugoslavia</a></li>\n",
      "<li><a href=\"/wiki/History_of_Germany_(1945%E2%80%931990)\" title=\"History of Germany (1945–1990)\">Partition of Germany</a></li>\n",
      "<li><a href=\"/wiki/Territorial_changes_of_Poland_immediately_after_World_War_II\" title=\"Territorial changes of Poland immediately after World War II\">Borders of Poland changed</a>.</li></td></tr></tbody>\n",
      "----------------------------\n",
      "<table style=\"width:100%;margin:0;padding:0;border:0\"><tbody><tr><th style=\"padding-right:1em\">Date</th><td>22 June 1941<span style=\"display:none\"> (<span class=\"bday dtstart published updated\">1941-06-22</span>)</span> – 25 May 1945<span style=\"display:none\"> (<span class=\"bday dtstart published updated\">1945-05-25</span>)</span><sup class=\"reference\" id=\"cite_ref-2\"><a href=\"#cite_note-2\">[2]</a></sup><br/>(3 years, 11 months and 3 days)</td></tr><tr><th style=\"padding-right:1em\">Location</th><td><div class=\"location\">Europe east of Germany: <a href=\"/wiki/Central_and_Eastern_Europe\" title=\"Central and Eastern Europe\">Central and Eastern Europe</a>, in later stages Germany and Austria</div></td></tr><tr><th style=\"padding-right:1em\">Result</th><td>\n",
      "<p>Soviet victory\n",
      "</p>\n",
      "<ul><li>Fall of the <a class=\"mw-redirect\" href=\"/wiki/Third_Reich\" title=\"Third Reich\">Third Reich</a></li>\n",
      "<li><a class=\"mw-redirect\" href=\"/wiki/Allied_occupation_of_Germany\" title=\"Allied occupation of Germany\">Allied occupation of Germany</a></li>\n",
      "<li>Beginning of the <a href=\"/wiki/Cold_War\" title=\"Cold War\">Cold War</a> and creation of the <a href=\"/wiki/Eastern_Bloc\" title=\"Eastern Bloc\">Eastern Bloc</a> and the <a href=\"/wiki/Iron_Curtain\" title=\"Iron Curtain\">Iron Curtain</a></li>\n",
      "<li>Beginning of the <a href=\"/wiki/Greek_Civil_War\" title=\"Greek Civil War\">Greek Civil War</a></li></ul></td></tr><tr><th style=\"padding-right:1em\">Territorial<br/>changes</th><td>\n",
      "<li>The Soviet Union occupies <a href=\"/wiki/Central_Europe\" title=\"Central Europe\">Central</a>, <a href=\"/wiki/Eastern_Europe\" title=\"Eastern Europe\">Eastern</a>, <a class=\"mw-redirect\" href=\"/wiki/Northeastern_Europe\" title=\"Northeastern Europe\">Northeastern</a> and <a class=\"mw-redirect\" href=\"/wiki/Southeastern_Europe\" title=\"Southeastern Europe\">Southeastern Europe</a> and establishes pro-Soviet <a href=\"/wiki/Communist_state\" title=\"Communist state\">communist</a> <a href=\"/wiki/Puppet_state\" title=\"Puppet state\">puppet governments</a> in countries including <a href=\"/wiki/People%27s_Republic_of_Bulgaria\" title=\"People's Republic of Bulgaria\">Bulgaria</a>, <a href=\"/wiki/Czechoslovak_Socialist_Republic\" title=\"Czechoslovak Socialist Republic\">Czechoslovakia</a>, <a href=\"/wiki/Hungarian_People%27s_Republic\" title=\"Hungarian People's Republic\">Hungary</a>, <a href=\"/wiki/Polish_People%27s_Republic\" title=\"Polish People's Republic\">Poland</a>, <a class=\"mw-redirect\" href=\"/wiki/Romanian_People%27s_Republic\" title=\"Romanian People's Republic\">Romania</a>, and <a href=\"/wiki/East_Germany\" title=\"East Germany\">Eastern Germany</a>.</li>\n",
      "<li>Establishment of the <a class=\"mw-redirect\" href=\"/wiki/Federal_People%27s_Republic_of_Yugoslavia\" title=\"Federal People's Republic of Yugoslavia\">Federal People's Republic of Yugoslavia</a></li>\n",
      "<li><a href=\"/wiki/History_of_Germany_(1945%E2%80%931990)\" title=\"History of Germany (1945–1990)\">Partition of Germany</a></li>\n",
      "<li><a href=\"/wiki/Territorial_changes_of_Poland_immediately_after_World_War_II\" title=\"Territorial changes of Poland immediately after World War II\">Borders of Poland changed</a>.</li></td></tr></tbody></table>\n"
     ]
    }
   ],
   "source": [
    "# just for speed purposes, let's only do a few parents\n",
    "par_list = list(the_parents)\n",
    "\n",
    "for parent in par_list[0:2]:\n",
    "    print('----------------------------')\n",
    "    print(parent)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
